LOCAL CHANGE OF AN IMAGE SHARPNESS OF PHOTOGRAPHIC 
IMAGES WITH MASKS 



5 

Field Of The Invention 

The present invention relates to a process for the change 
of an image sharpness of a photographic image by way of a 
mask. It further relates to a corresponding device, program, 

10 computer program medium with the program and image output 
apparatus, such as photographic printers, or printers or 
photographic labs or monitors (for example LCD or CLT display) 

M or computers with monitors, which use the process or the 

'III program or the device. 

"si Image sharpening processes are known, for example, from A. 

m 

sj: Polesel, G. Ramponi, "Image Enhancement via Adaptive Unsharp 
■4 Masking", IEEE Trans. Image Processing, Vol. 9, No. 3, Pgs . 

505-510, March 2000, or N. Arad, C. Gotsman, "Enhancement by 
£p Image-Dependent Wrapping", IEEE Trans. Image Processing, Vol. 
2& 8, No. 8, Pgs. 1063-1074, August 19, 1999. The image 
;|p| sharpening processes are based on a determination of the image 
^ sharpness (the contrast) at a location at which the image 

sharpening is to be carried out. The contrast (the sharpness) 

is herefor calculated based on brightness data of the 
25 neighbouring image elements, in that differences between the 

different brightness data are calculated according to certain 

rules . 

Summary Of The Invention 
30 It is an object of the invention to enable a local image 

sharpening of photographic image data in order to globally 
improve the image. 

This object is achieved with a process in accordance with 
the invention which uses (especially digital) masks for the 
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image sharpness correction. A mask hereby means a correlation 
between values (which are herein referred to as elements of 
the mask) and parts of the image, whereby these image parts 
represent a location in the image and, for example, include an 
5 image element (pixel) or several image elements (pixels) . 
Thus, the elements of the mask represent a rule for the local 
changing of the image sharpness of the image in a certain 
image portion. They describe especially the degree or extent 
of the image sharpening to be carried out, especially a 
10 contrast increase or decrease at the respective locations, at 
which the image sharpening is carried out. 

The global contrast, in the following also simply referred 
Mi; to as "contrast", particularly corresponds to the difference 

%% (and/or the ratio) between maximum and minimum brightness in 

w 

15 the (whole) image. The global contrast is preferably 
Sj| determined based on an image which was subjected to an image 
*j detail removal process. Preferably, the global contrast is 
a determined based on an image with reduced resolution. The 
advantage hereby resides in that the sensitivity to noise and 
20 to unimportant image details becomes lower. 

\j. The detail contrast, which in the following is also called 

j ^ local contrast, describes especially, the difference (or the 
ratio) between maximum and minimum brightness in a certain 
area surrounding an image point. Preferably, the detail 

25 contrast again is determined based on an image which was 
subjected to a detail removal process and especially an image 
which has a reduced resolution. 

The image sharpness or focus is principally determined by 
the resolution capacity of the camera optics. This is 

30 characterized, for example, by the modulation transfer factor 
(as a function of the local f reguency) , for example. An 
inexact focussing causes an additional lack of focus. When the 
depth focus is missing, not all objects of the scene can be 
imaged egually in focus. The focus is preferably judged on the 
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basis of the image with high resolution. Accordingly, ' the 
correction mask for the change of the image sharpness 
described in the following is preferably applied to the high 
resolution image (fine image) . As a measure for the lack of 
focus and thus the lack of image sharpness, a number of pixels 
can be used, for example, over which an edge in focus is 
"smudged". The image sharpness impression can preferably be 
improved, for example, by filtering (with a filter which 
enhances higher local frequencies) . 

In accordance with the invention, the global change of the 
image sharpness is carried out especially in dependency of the 
local contrast. 

The elements of the correction mask for the change of the 
image sharpness locally describe especially the degree of the 
contrast changes to be carried out for the individual image 
elements, which means especially the change of the local 
contrast (detail contrast) . Especially preferably, the 
elements of the correction mask for the change of the image 
sharpness describe the change of the image sharpness which is 
carried out by way of an adaptation process. 

For determination of the elements, information on an image 
property is especially used, especially a local image property 
(especially the image property at the location associated with 
the element) . The image property is preferably the local 
contrast (detail contrast) at the location associated with the 
element (or the local brightness) . In accordance with the 
invention, the elements are determined not only based on an 
image property, but further information which relates to the 
image is used for the determination of the elements. 

The process in accordance with the invention preferably 
uses, for the image sharpening, an image sharpness adaptation 
process, whereby the elements of the correction mask for the 
change of the image sharpness fix the change of the image 
sharpness produced by the image sharpness adaptation process. 



The image sharpness change generally can take place so 
that the image sharpness (local) is approximated to a nominal 
image sharpness value or a range for the nominal image 
sharpness, or that it reaches the nominal image sharpness 
5 value or lies within the nominal image sharpness value range. 

The present invention relates to correction masks for the 
change of the image sharpness which influence an image 
property, namely the image sharpness. An image property is an 
appearance property of the image observed by a human observer 
10 and especially of image regions, such as brightness, colour 
tone, colour saturation, intensity of a colour (for example 
the value of R for an RGB vector, or the value of C for a CMY 
\,& vector) , image sharpness, contrast, average brightness, 
average colour, relative number of points of a characteristic 
} : 5 colour (for example "memory colour"), local contrast 
%| (difference between maximum and minimum brightness) , fine and 
average detail contrast, histogram of the brightness values 
s (feature vector) , colour histogram (mermal matrix) mean colour 
pjj saturation, brightness gradient left/right, up/down, and so 
i|| on. Image properties are described, for example, in R.W.G. 

Hunt, "Colour Reproduction by Photography" Rep. Prog. Phys . 
■V 1977, 40, pgs. 1071-1121. 

In accordance with the invention, the correction mask for 
the change of the image sharpness is determined from the 
25 photographic (especially digital) image data (for example, 
original image data) to be corrected or image data derived 
therefrom. 

The elements of the mask for the change of the image 
sharpness are called "Correction Elements". The correction 
30 elements are associated with locations in the image, which 
relate to parts of the image. Such an image part can 
correspond to an image element (pixel) , whereby an image datum 
is associated with exactly one pixel. However, the image part 
can also include several image elements. The correction 
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elements set the correction for those image data which relate 
to the image parts and fix the image property therefor. The 
correction elements set the change in the image sharpness in 
the image parts with which they are associated. 
5 In order to achieve the correction, the correction mask is 

applied to the image data. A image sharpness adaptation 
process is used herefor, for example an adaptive high-pass 
filtering or warp-sharpening, in which the extent of the local 
for the change of the image sharpness in an image part is 
10 carried out independent of the mask element which is 
associated with this image part. A suitable colour space is 

f>fc preferably used for this correlation, whereby before and/or 

'm 

t% after the correlation a transformation can be carried out in 
! 'i another colour space. 

%!§5 According to one embodiment, the correction mask for the 

;*| change of the image sharpness is determined not only 
h independent of one image property (for example detail 
Spe contrast), but in dependency of several image properties. 
.\ It is an important aspect of the invention that the 

:|0 correction masks for the change of the image sharpness are 
constructed such that detail information contained in the 
image is not lost. For this purpose, the elements of the 
correction mask are preferably designed such that they correct 
adjacent image elements as equally as possible, or with only 
25 little difference. To achieve this, different methods can be 
used. For example, the image data representing the image can 
be subjected before and/or after the correction to a process 
which causes a softening, such as, for example, a data 
reduction process (for example Downsampling process) , a low- 
30 pass filtering, an averaging process or a gauss filter. 
Furthermore, with these processes used for the removal of 
image details images can be produced which represent the 
original image without details, i.e. "coarse". Such images are 
in the following referred to as "coarse images". Based on 
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these coarse images, unsharp masks can be produced. These 
masks which cause a less sharp image are preferably calculated 
on the basis of a low-resolution image (coarse image) which 
was achieved, for example, by Downsampling, whereby especially 
5 processing capacity can be saved. Images which include the 
image details, i.e. those which are not subjected to a detail 
removal process, are in the following referred to as "fine 
image" . The image data representing the fine image are 
referred to a fine image data. The correction masks can be 
10 determined directly from the fine image data. However, for the 
reasons mentioned above, they are preferably based on coarse 
image data which are derived from the fine image data (for 
W example original image data) . Correspondingly, the coarse 
pi image is derived from the fine image (original image) . 
H% The coarse image is represented by coarse image data, 

%| which result from the detail removal process. The coarse image 
,'1 r just like the original image (fine image) has a multitude of 
image elements which are in the following referred to as 
m "coarse image elements" whereby "coarse image portions" can be 
20 the same as a coarse image element or may include several 
;p coarse image elements. A correction mask which is produced on 
the basis of coarse image data is called coarse correction 
mask, the elements or which relate to the coarse image 
portions. The elements of the coarse image mask describe 
25 changes in the contrast of the coarse image which are referred 
to as "coarse changes" and which set the change of the image 
properties or at least one image property of the coarse image 
portions. The coarse changes are designed with a view to the 
contrast changes which must be carried out by the correction 
30 mask. For example, the elements of the coarse correction mask 
are designed such that they change the contrast of the coarse 
image. When the coarse image was produced with a data 
reduction process, the correction mask can then be produced 
from the coarse correction mask by an expansion process 
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C x upsampling" ) . Should the number of the coarse image elements 
correspond to the number of the fine image elements, which 
means the coarse image includes less details but no data 
reduction process was used, then the correction mask can be 
5 equated to the coarse correction mask. 

In accordance with the invention, the contrast change 
fixed by the elements is a function of the local contrast, 
i.e. the contrast at the location to which the mask element 
relates, as well as a function of additional information which 
10 relates to the image. The additional information can be 
obtained from the image data (for example information on the 
colour tone, or image regions which represent skin) as will be 
b& described further below, or can be input into the process in 
i^i accordance with the invention or the system in accordance with 
% the invention in addition to the image data or can be received 
%| by them. For example, it can be information on the situation 
i'^ during the image capturing (use of a flash, camera 
adjustments, etc. so on) or information on the image capturing 

si 

Pjj apparatus (camera) . The latter relates especially to 

20 information on the camera, the film type and image data 
processing processes (for example meta-inf ormation on the 
image compression, for example with JPEG) which were carried 
out before and so on. 

The additional information regarding the image makes it 

25 possible to carry out a image sharpening adapted to the 
characteristics of the image. 

According to one embodiment, the additional information is 
information on local image properties which is obtained by 
analysis of the image data and which is available in addition 

30 to information on the local contrast (and/or the local 
brightness) . The additional local image property can be the 
colour tone and/or the colour contrast and/or the colour 
saturation which is locally present, which means in the image 
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element or the image elements to which the elements of the 
correction mask relate or in the vicinity of those elements. 

The contrast change carried out with the correction mask 
for the change of the image sharpness is especially a change 
in the brightness contrast. The brightness contrast is a 
function of the difference of the brightness values of 
neighbouring image elements. 

A change caused by the correction mask can also be a 
change of the colour contrast. The colour contrast is defined, 
for example, as a function of the distance of the colour 
values from neighbouring image elements in the colour plane 
(a,b plane) . The colour contrast can also be defined as a 
function of the differences of the colour intensity of 
neighbouring image elements. For example, the colour contrast 
for red can be defined as a function of the differences of the 
R values (of an R, G, B-vector ) of neighbouring image elements. 
A correction mask which corrects the colour contrast of a 
specific colour therefore preferably acts only on that aspect 
of the image data which represents this colour (for example 
the R value) , whereby the correction mask, which causes a 
change of the brightness contrast, preferably influences that 
aspect of the image data which represents the brightness (for 
example the L value of a Lab vector) . 

The colour contrast can be changed differently for 
different colours. For example, a correction mask exists which 
changes the colour contrast for the colour red, another 
correction mask which changes the colour contrast for the 
colour green and finally a further correction mask which 
changes the contrast for the colour blue. Of course, 
correction masks for other colours are also possible (for 
example C, M, Y) . They are preferably independent colours 
which span a colour space. By adaptation of the correction 
mask for the change of the image sharpness to the additional 
local image properties, an advantageous image sharpening can 



be achieved. For example, the contrast for typical colour 
tones (for example skin tones) can be rather reduced while the 
contrast is rather increased for other typical colour tones 
(for example vegetation green) . Also, in regions of low colour 
5 saturation, it can be desirable to lower the contrast in order 
to avoid possible noise, while it can be desirable in regions 
of high colour saturation to increase the contrast to make 
structures more clearly apparent. 

According to a further embodiment, the additional 
10 information additionally or alternatively is information on 
the image content. This information can be obtained through 
additional data which were received in addition to the image 
^ data. For example, the information can be transmitted that, 
the portrait picture type was adjusted at the camera during 
% the image capture. Accordingly, it can be assumed that the 
y image content includes regions which show skin. If the 
'1 additional data indicate that a flash was used, it can be 
assumed that with a certain probability overexposed and 
underexposed regions exist in the image. The images can 

'$0 therefore be assigned to different classes on the basis of the 

% 

■f t additional information, whereby then a determination of the 

k elements of the correction mask is carried out depending on 
the classes or the already determined elements are subjected 
to a class dependent correction. 

25 Preferably, the determination of the additional 

information on the image content is carried out based on an 
analysis of the image data, preferably at least on the basis 
of a majority of the image data. The image can then be 
classified on the basis of the analysis of the image data. 

30 Depending on the class (for example sunrise, flash 
photography, portrait photograph, and so on) the correction 
mask for the change of the image sharpness is determined. 

The image data are preferably analyzed in order to 
recognize characteristic image regions. Characteristic image 
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regions are image regions which appear frequently in 
photographic images. They include for example regions which 
illustrate a face or skin, regions which illustrate 
vegetation, for example, trees, regions which illustrate 
5 buildings or streets, regions which illustrate sky or beach, 
and so on. The characteristic image regions can but need not 
consist of neighbouring image elements. Generally, the 
majority of the image elements of a characteristic region are 
neighbouring elements. 
10 Such regions can be determined, for example, by pattern 

recognition, for example, for face recognition or by 
k& recognition of structures. For example, face recognition is 
T possible by way of pattern recognition processes. Information 
;* on colour tones is not necessarily required herefor. 
*-\p Preferably, information on the colour tone is used for 

,'jf the recognition of characteristic regions. For example, skin, 
sky or vegetation has a typical characteristic colour or 
*& typically occupies a certain region in the colour space. By 
t static analysis of a plurality of images one can determine, 
0) for example, with which probability a certain colour tone 
1 ^ belongs to a certain characteristic colour. 

When a characteristic image region is recognized, the 
determination of the correction mask for the change of the 
image sharpness is carried out based on this recognition. The 
25 determination of the correction mask can be carried out, for 
example, in that, in a first step, the correction mask is 
determined on the basis of the local contrast or the local 
brightness and in a second step the elements of the correction 
mask are changed in such a way that they at least approximate 
30 a nominal image sharpness or a nominal image sharpness range, 
which is preselected for the recognized characteristic image 
region. Preferably, an association rule is provided which 
associates the nominal focus or the nominal image sharpness 
ranges with different types of characteristic image regions. 
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The determination of the correction mask for the change 
of the image sharpness can thereby be additionally carried out 
on the basis of a degree of belonging. The degree of belonging 
describes the degree of belonging of an image element to a 
5 characteristic image region. For example, in the above 
mentioned example, the degree of belonging describes the 
degree of the probability with which a new colour belongs to a 
characteristic colour tone and thereby to a characteristic 
image region. The correction of a mask determined in a first 
10 step is then carried out depending on the degree of belonging 
in the direction of the nominal image sharpness. This means 
that the larger the degree of belonging, the further the 
approximation to the nominal image sharpness. 

in 

£ The inventors of the present application have discovered 

S% that at the transition between characteristic image regions a 
%j change of contrast, especially an image sharpening is critical 
.'1 and can easily lead to artefacts. Thus, in accordance with the 
invention, the image, or an image derived therefrom (for 
;p example a coarse image) is examined for transitions between 
® image regions which have a different structure, which means, 
if?p for example, if high or low space frequencies dominate or if 
■ ^ one image property of one image region differs significantly 
from the same image property of the neighbouring image region 
(for example by a preselected threshold value or factor) . Such 
25 transitions exist, for example, between forest and blue sky. 
Such a transition can be detected apart from an analysis of 
the space frequencies, for example, by analysis of the 
distribution of the contrast values. In highly structured 
regions, a high mean detail contrast appears, while the mean 
30 detail contrast is low in less structured regions. Transition 
regions can be localized, for example, where the first 
derivative of the contrast course takes on extreme values. In 
accordance with the invention, preferably no or only a weak 
contrast change is carried out in the transition regions, to 
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not provoke artefacts. This is the case especially for regions 
with high detail contrast, which means if the detail contrast 
is above a threshold value. Alternatively, or additionally, 
the course of the contrast values after a carried out image 
5 sharpening can be subjected to a smoothing by the correction 
mask in the transition region. The transition between weakly 
structured and strongly structured regions is preferably 
considered only with compressed images, especially preferably 
only with block-compressed images ( DCT-compression, JPEG) 
10 during the determination of the mask in the above described 
manner, since (too high) image sharpening strengthens 
compression related artefacts ("ringing"). "Ringing" is 

'y* generated, for example, when coefficients of transformed 
images are quantified or the number of coefficients is 
)5 reduced. Thus, especially in those compression processes and 
%| in loss associated coding processes ("lossy coding) , the 
/•{ transitions are considered. The consideration is preferably 
carried out in such a way that no focussing takes place where 
S|J the image sharpness is reduced (unf ocussing) . 

20 If the image data are present in the form of coefficients 

;?| which determine the weighting of different space frequencies, 
! ^ as is the case, for example, with the coefficients which 
result from a discreet cosign transformation (DCT) , as is the 
case with the image compression process JPEG, it can be 
25 recognized by way of the coefficients where strongly or weakly 
structured regions are present. Strongly structured regions 
are thereby regions especially with a high proportion of high 
space frequencies and/or a high contrast and weakly structured 
are especially regions with a low proportion of high space 
30 frequencies and/or with a low contrast. A transition region 
can then be recognized from an analysis of the coefficients. 

If it is known from the additional information, which is 
transmitted in addition to the image data, that the image data 
were subjected to a process which leads to block formation, as 
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is the case, for example, with JPEG, where the image is 
blockwise compressed (blocks with 8x8 image elements) , the 
block transitions are preferably especially treated. 
Particularly, the correction mask (preferably based on the 
5 fine image) is constructed such that the mask elements which 
relate to the block transitions or block edges, cause no or 
only a small change in the contrast. The block edges are 
phantom structures which will be strengthened throughout the 
image without the consideration of the additional information. 

10 Generally, in accordance with the invention, it is 

preferred that any additional information is processed from 

L which the position of artefacts in the image can be deduced. 

Q- One preferably foregoes an image sharpening at the position of 

? the artefacts. 

15 The present invention further relates to a device (for 

%| example hardware and software or ASIC) for the image 
sharpening of a photographic image. This preferably includes a 

% correction mask determining unit, for example a CPU, which, 

Q for example, stores the image of a correction mask in a memory 
at preselected addresses, in order to guarantee that the 

fjj elements can be associated with image parts. The correction 
mask determining unit determines from the image data which 
represents the image to be focussed, or from image data 
derived therefrom (for example, coarse data) the elements of 

25 the correction mask. The elements thereby fix the extent of 
the image sharpening to be carried out for the individual 
image elements (or coarse image elements) . The device 
preferably further includes an image property determining unit 
(for example a contrast /brightness determining unit) which 

30 locally determines an image property (for example a contrast) 
in the image (or coarse image) . Based on the determined 
values, the determination unit calculates the mask elements. 
Preferably, the device further includes an additional 
information processing unit, which processes further 
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information relating to the image and influences on the basis 
thereof the determination of the mask elements by the 
determining unit. The image sharpness change in accordance 
with the invention which is carried out by an image sharpness 
5 change unit depending on the elements of the correction mask, 
preferably includes the change of a brightness contrast and/or 
the change of at least one colour contrast. 

The invention further relates to a program which induces 
a computer to carry out the process in accordance with the 
10 invention as well as a computer program storage medium on 
which the program is stored. 

The present invention further includes image output 
h& devices, for example, photographic printers or printers (for 
M example, inkjet printers, CRT printers, DND printers, 
;: ;t| photographic printers, monitors, CRT displays, LLD displays) 
%% and photographic labs, especially minilabs and microlabs, but 
also large labs which use the process in accordance with the 
* invention, the device in accordance with the invention or the 
% program in accordance with the invention. 

JJj Brief Description Of The Drawings 

Further features of the invention are disclosed in the 

following description of detailed embodiments, whereby 

features of different embodiments can be combined with one 
25 another. The invention will be further described in the 

following by way of example only and with reference to the 

drawings wherein 

Figure 1 shows an exemplary seguence of a local focus 

control according to the invention; and 
30 Figure 2 illustrates the incorporation of the local focus 

control in accordance with the invention into a general 

correction system. 
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Detailed Description Of The Preferred Embodiments 

In accordance with the invention, the additional 
information relating to the image is preferably captured in 
the form of masks which are then used for the correction of an 
5 already determined correction mask for the change of the image 
sharpness. This results in a clear association between the 
elements of the correction mask for the change of the image 
sharpness and the additional information mask for the 
optimization of the local image sharpness correction 
10 properties of the process in accordance with the invention. 

For example, skin, sky and vegetation masks are used, 
h& which represent the additional information and which serve the 
K calculation of the correction mask for the change of the image 
;jf sharpness or the correction of a correction mask for the 
: -f> presharpening of the image calculated in the first step. 

It is described in the following by way of example how a 
^ correction mask for the change of the image sharpness is 
f| determined. The description is with reference to Figure 1. 

First, a contrast mask is formed, which characterizes the 
ijfp course of the local contrast in the image. The analysis 
required therefore is preferably carried out on the basis of a 
coarse image with an average image resolution (typically 200 x 
250 pixel) . The contrast mask so obtained corresponds to the 
above mentioned coarse mask. The average image resolution is 
25 preferably by a factor of 3 to 20 smaller than the resolution 
of the original image and is produced from the original image 
by a data reduction process, for example, by " down samp ling" , 
whereby image data of the coarse image represent mean values 
of several neighbouring image data of the fine image. The mean 
30 image resolution is preferably about 20,000 to 200,000 pixels. 
An original image typically includes over 1,000,000 pixels and 
typically between 1,000,000 and 5,000,000. The difference 
between maximum and minimum brightness in a vicinity of the 
neighbouring image point can be used, for example, for a 
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measure of the detail contrast. The detail contrast is a local 
image property and preferably describes neighbouring points 
within a small surrounding area, for example 5x5, 7x7, 9x9 
pixel, and so on. However, the surrounding area could also be 
5 larger, for example, 100x100 pixel. However, as described 
above, a colour contrast can also be determined and taken into 
consideration. 

The "contrast mask" determined in this way is now 
recalculated by way of a function into a "base image 
10 sharpening mask". The function is preferably nonlinear. The 
function can be defined through a table (LUT) . The elements of 
the base focussing mask represent factors for the change of 
b& the image sharpness and determine the degree of image 
sfj sharpening. The connection between the image sharpening 
'"4s factors and the local contrast is shown in Figure 1. 

m 

\% The image sharpening factors of the base focussing mask 

preferably have the following properties: 
n • The image sharpening factor decreases in the region of 

~' v smaller detail contrast. This avoids that the noise is 

20 increased in underexposed regions and that homogeneous 

;3 image regions (such as, for example, sky) have a "grainy"' 

appearance. 

The image sharpening factor preferably monotonously 
decreases to 0 in the regions of higher detail contrast. 
25 It is achieved thereby that already image sharpened image 

regions are not over-sharpened and that in the region of 
sharp edges no disturbing overswings are created. 

The function preferably has a maximum in the region of 
average detail contrast values which are at about 10-50%, 
30 preferably about 20-40% of the maximally determined 

contrast . 

The function, especially the nonlinear function, is 
preferably adapted to the output medium and optimized in 
such a way that a maximum image sharpening without 
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artefacts is achieved. This is preferably achieved by way 
of test patches. Test patches are test images to which 
the correction mask for the change of the image sharpness 
is applied which was determined based on the base image 
sharpening mask. The quality of the correction mask for 
the change of the image sharpness and thereby the 
nonlinear function can be determined by visual or 
automatic analysis of the test patches. The nonlinear 
function can therefore be optimized by iterative 
processes. Test patches are generally made of a strip 
pattern, whereby the strips have different detail 
contrasts (brightness levels) . 



The base image sharpening mask is preferably a coarse 
mask as with the contrast mask, which means is based on coarse 
image data, which do not contain detailed information. The 
coarse image data are determined from the fine image data by a 
detail removal process. If a data reduction process (for 
example w downs ampling" ) is used as the detail removal process, 
the coarse masks have a reduced number of elements, which 
preferably corresponds to the number of elements of the coarse 
image. The base image sharpening mask, however, is preferably 
not necessarily present with a reduced number of elements. The 
base image sharpening mask in a following process step is now 
corrected in order to obtain the image sharpening mask in the 
end. The correction is again preferably carried out with a 
reduced number of elements. In that case, the corrected base 
image sharpening mask must then be subjected to an expansion 
process (for example interpolation process) , in order to 
obtain a correction mask for the change of the image sharpness 
with a number of elements which corresponds to the number of 
elements of the fine image to be corrected. 

The base image sharpening mask is preferably corrected 
based on other masks, so called additional information masks 
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or auxiliary masks (see Figure 2) . An element of the 
additional information mask is thereby preferably clearly 
associated with an element of the base image sharpening mask 
and describes the degree of change of the associated element 
5 of the base image sharpening mask. The additional information 
masks (auxiliary masks) are preferably determined based on a 
coarse image (for example, the one used as basis for the local 
contrast analysis, or based on an analysis of this coarse 
image) . The elements of the additional information mask, for 
10 example, can be a function of the local brightness or the 
local colour saturation or the local colour tone. Preferably, 
^ the additional information masks represent information on the 
!jf image content, such as information on characteristic image 
%| regions, such as skin or face. The additional information mask 
: 15 is in this case also referred to as skin mask. Other examples 
s 4 for additional information mask are the sky masks and the 
t vegetation masks illustrated in Figure 1. 

The image sharpening factor is preferably reduced in 
i!J image regions with skin colour. The fact is thereby considered 
2%Q that with excessive image sharpening, the impression of 
W "unclean" skin is created (pimples, beard hair) . Consequently, 
this reduction can be carried out by way of the multiplication 
of the image sharpening factor with the expression (1.0 - k x 
skin indicator), whereby k<l, preferably k=0 . 5 . The skin 
25 indicator is a measure for the probability or belonging of an 
image element to a characteristic image region illustrating 
skin. The value of the skin indicator of an image element can 
be determined, for example, on the basis of the colour value, 
especially the colour tone and/or the colour saturation of the 
30 image element. Preferably, distribution functions or 
probability distributions are given which establish with which 
probability or with which degree of belonging a certain colour 
value belongs to a characteristic colour (for example skin 
colour) . The more a colour value deviates from the 
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characteristic colour value, especially a colour tone and/or 
colour saturation, the smaller is typically the degree of 
belonging. From this distribution function or probability 
function a measure can be determined on the basis of the 
5 colour values, especially the colour tone and/or the colour 
saturation, for whether a specific colour value belongs to a 
characteristic colour. This skin indicator is then determined 
therefrom, for example. 

• In image regions with sky colour, the image sharpening 
10 factor is preferably also reduced according to an analog 

process or even reduced to 0. 

• In image regions with vegetation, a stronger image 
f»* sharpening is genuinely carried out. This can be achieved 
|>fi analog to the above by way of the vegetation mask. 

;i5 • Finally, the corrected base image sharpening mask 

\| determined under consideration of the additional 

information is brought to the resolution of the image to 
ft be sharpened (fine image) by way of interpolation 

("upsampling" and "smoothing"), which means a data 
30 expansion process. 

• The original image {fine image) is then subjected to a 
^* image sharpness change process which uses as parameters 

the elements of the correction mask for the change of the 
image sharpness. The image sharpness change process can 

25 be for example, a location variable high pass filtering, 

as described in A. Polesel, G. Ramponi, "Image 
Enhancement via Adaptive Unsharp Masking", IEEE Trans. 
Image Processing, Vol. 9, No. 3, Pgs . 505-510, March 
2000. This means the degree of image sharpening varies 

30 from image element to image element on the basis of the 

values of the elements in the image sharpening mask which 
are especially clearly associated with the image 
elements. For example, the increase of the higher local 
frequencies during the high pass filtering process (for 
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example elevation of the strengthening factor of the 
filter and/or change of the limit frequency of the 
filter) can be changed depending on the elements of the 
correction mask for the change of the image sharpness. If 
5 an unsharpening is desired, the higher local frequencies 

can be lowered, for example. The increase or reduction of 
the image sharpness is generally referred to as image 
sharpness change. The variation of the image sharpness is 
thereby gradual, since the coarse image forms the basis 
10 for the correction mask for the change of the image 

sharpness, which includes no detail information. The 
correction mask for the change of the image sharpness can 
h& thereby be considered a low frequency (which means 

?4 unsharpened) mask. A process for the change of the local 

15 contrast can preferably be used as image sharpness change 

%| process, whereby the change of the contrast depends on 

I \ the elements of the correction mask for the change of the 

-« image sharpness. 

W- 

20 As already mentioned above, several possibilities exist 

for the carrying out of the image sharpening locally depending 

m 

on the correction mask. At the adaptive high pass filter, the 
values associated with the pixels are changed. In contrast, 
the warp sharpening in an intermediate step leaves the pixel 

25 values unchanged. A displacement of the pixels results 
instead. Overswings can be avoided in this manner. The 
functioning of the warp sharpening is described in the above 
mentioned reference by N. Arad, C. Gotsman. Candidates for 
edges are thereby searched in the image by way of an edge 

30 detector. The output of the edge detector is then softened. 
Thereafter, a relief filter is used and a displacement mask 
derived therefrom. The original pixels are then displaced by 
the value given in the displacement mask. This can be carried 
out, for example, by the principle that upon the displacement 
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by more than half a pixel, the values of the displaced pixels 
replace the values of the pixels which so far were at this 
location. Where locations would be vacated by the displacement 
process, the old values are maintained, for example. The 
5 amount of displacement can, for example, be carried out 
depending on the elements of the image sharpening correction 
mask. 

Since the pixels in such a image sharpening process are 
always displaced towards the edge, the edges are steepened 
10 (made steeper) whereby a image sharpening is achieved. An 
unsharpening of the image would be achieved by displacing the 
pixels in such a way that the edges are made flatter. 

As already mentioned above, the coarse image can be 
achieved by downsampling or with filter processes (low pass 
[% filter, gauss filter) or by filter banks (as described in EP 0 
971 315) . A downsampling in a first step and an upsampling at 
s ' p the end of the process is therefore not absolutely necessary, 
but is preferred because of the therefor required low 
processing capacity and the low storage cost. 
20 Figure 2 illustrates the incorporation of the local image 

51 sharpening process in accordance with the invention into a 
mask correction system which is further described in the 
following. 

Figure 2 shows an exemplary construction for a correction 
25 device in accordance with the invention, whereby the blocks 
respectively represent units which carry out certain process 
portions. Alternatively, Figure 2 can also be considered as 
illustration of the correction process in accordance with the 
invention. Figure 2 especially shows the data flows and the 
30 co-operation of the individual components of the system. 

Reference numerals which in Figure 2 commence with 1 
relate to the lowest resolution. Reference numerals which 
begin with 2 relate to the mean resolution and reference 
numerals which begin with 3 relate to the highest resolution. 
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Reference numerals which commence with 4 relate to a lowering 
of the resolution and reference numerals with commence with 5 
relate to an increase in the resolution. The broken lines in 
the upper portion of the image relate to the lower resolution. 
5 The thick continuous lines in the lower part of the image 
relate to the higher resolution. 

According to the example shown in Figure 2, an image 
analysis preferably takes place with the low resolution 
images. The analysis of the local contrast in block 210 
10 represents an exception. The reduction of the image resolution 
(" downsampling" ) is carried out preferably in at least two 
steps. In the first step, which is carried out in block 410, a 
reduction occurs of the resolution from the high resolution 
image (typically over 1,000,000 image elements) to a mean 
15 resolution, for example, 200x250 image elements. In block 420 
v occurs in a second step a further lowering of the image 
/jf resolution to a low resolution, for example 40x50 image 
a points. The analysis can thereby, if necessary be carried out 
| on images of different resolution. 

20 The coarse masks which are used in blocks 110, 120 and 

© 

Q 130, are determined based on an analysis of the coarse image, 

mi; 

' w which results from block 420. The coarse masks are converted 
into unsharpening masks (correction masks) which preferably 
correspond to the resolution of the original image, which 
25 means one element of the correction mask corresponds to an 
image element of the original image or the fine image to be 
corrected. 

The increase in resolution is preferably carried out in 
the same number of steps as the reduction of the resolution so 
30 that masks calculated with different resolution can be 
combined with one another (for example weighted) , even before 
one proceeds to the next resolution or next higher resolution. 

The sequence of the processing steps which are shown in 
Figure 2 and which are carried out on the image data can also 
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be differently selected. The illustrated processing chain only 
serves for the illustration of the concept. For reasons of 
efficiency, it is naturally sensible to combine the individual 
processing steps as much as possible. 
5 The blocks 110, 150 and 310 serve the purpose of using 

the available value range or dynamic range as much as possible 
and are described in more detail further below. The remaining 
elements of Figure 2 are first described in more detail in the 
following. 

10 At the start, a high resolution image with a multitude of 

image data which are respectively associated with image 
k& elements is received (in the image at the bottom left) . The 
ft image data are reduced in a first step in block 410 and then 
V again reduced in the block 420. The reduced image data present 
sj$ after the block 420 therefore represent an image of low 

* resolution (coarse image) . The block 110 is applied to this 
J coarse image, which leads to a spreading of the value range as 
f|p will be described further below. A multiplying mask 120 is 

W then applied to the then resulting image data which are still 

4} 

:?0 present in the same resolution, which mask, for example, has 
itf the above mentioned effects on colour saturation. Finally, in 
block 130, an additive mask is applied which, for example, 
changes the brightness profile. Further masks can be applied 
thereafter which change the image properties. 

25 According to Figure 2, auxiliary masks are calculated 

based on the coarse image data in block 130. These auxiliary 
masks relate, for example, to characteristic image regions, 
such as skin, sky and vegetation, and can be determined, for 
example, as described above. By way of these masks, it is 

30 known to the system which image elements or image parts belong 
to the characteristic regions or which are associated with 
characteristic colour tones. 

The present invention especially relates only to the left 
portion of Figure 2, which means the blocks 420, 140, 560, 
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210, 510, 300, and 410. This portion can be combined in any 
way with other correction processes, especially with the right 
(remaining) portion of the Figure 2. 

A widening of the auxiliary masks to the mean resolution 
5 takes place in block 150. The information obtained therefrom 
is processed in the analysis block 210, which analyses the 
local image sharpness. The analysis of the local image 
sharpness is carried out on the basis of the mean resolution 
image. The reason for this is that the image sharpness at the 
10 position of the image element is defined as a function of the 
image property (for example brightness) at the location of the 
image element as well as at the location of neighbouring image 
hfr elements. A peculiarity of the process in accordance with the 
J?| invention resides not only in that not only local data flow 
% into the analysis of the local image sharpness, i.e. data 
%| relating to neighbouring image elements, but information on 
1; J the image content also flows in through the auxiliary masks 
s_ widened in block 560, and that depending thereon a mask is 
j==| determined, the elements of which identify a local image 
;;||) sharpening which considers respectively local data as well as 

O global image data (such as, for example, the belonging to a 

m 

characteristics image region) . The mask for the change of the 
image sharpness obtained in block 210 is then widened in block 
510. The widened correction mask 300 for the change of the 
25 image sharpness is then applied to the high resolution image 
data . 

The auxiliary masks are preferably used also for the 
calculation of the reference tables in block 510, which are 
then used for the expanding of the image data in block 110, as 
30 will be further described further below. The information 
included in the auxiliary masks especially on the image 
content is preferably also considered for the calculation of 
the coarse masks used in the blocks 120 and 130. The 
calculation of the multiplicative masks takes place in block 
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160 and the calculation of the additive masks in block 170. 
The calculation of the multiplicative and additive masks, i.e. 
in general of the coarse masks, is essentially carried out 
based on a coarse image and the auxiliary masks are preferably 
5 used for the correction of the coarse masks calculated this 
way. 

For example, the calculation of the multiplicative masks 
takes place in block 160 based on the analysis of the coarse 
image which results from the use of the LUTs in block 110, 
10 i.e. after expanding of the image data. The calculated 
multiplicative masks are then applied in block 120 to the 
coarse image. They lead, as discussed above, for example, to a 

h& change of the brightness in strongly overexposed or 

m 

underexposed regions and to a simultaneous change in the 
; i5 colour saturation in these regions. Brightness regions not yet 

m 

\^ corrected by the multiplicative masks can then in a next step, 
namely in block 130, be corrected by the use of additive 

m 

15 masks. The additive masks are thereby preferably calculated 

; ;;r; from the coarse image, which results in block 120 from the 

1 w 

Sto application of the multiplicative masks (or the multiplicative 
Q mask) . 

^ The coarse mask calculated in block 160 or the calculated 

coarse masks are then preferably transformed by expansion in 
block 520 into a mask of average resolution, i.e. with an 

25 average number of elements. In a further step, in block 540, 
the next expansion takes place which leads to a correction 
mask with a high number of elements. This correction mask is 
then applied in block 320 to the fine image data to be 
corrected, which result after the expansion in block 310. 

30 In an analogous manner, as with the multiplicative mask, 

a mask of average resolution is determined from the additive 
mask in two steps, namely in block 530, and then a mask with 
high resolution (correction mask) is determined through a 
further expansion in block 550. This correction mask which is 
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based on the coarse mask, is then applied in block 330 to the 
image data which result from the application of the 
multiplicative masks in block 320. 

The expansion to which the blocks 110, 150 and 310 

5 relate, will be described in the following. Block 150 relates 
to the calculation of the expansion. The block 110 relates to 
the application of the expansion to the coarse image and the 
block 310 relates to the application of the expansion to the 
image with high resolution. The expansion in block 310 is 

10 carried out with the help of a function with was determined in 
block 150. 

It is discussed in the following why such an expansion is 
^ preferably carried out. It must thereby be considered that the 
■y expansion of the dynamic region of the image data in the 
=«5 processing process shown in Fig. 2 is carried out in a certain 
%| portion of the process, namely after the application of the 
^ image sharpening of the image. The expansion can also take 

!* place elsewhere, for example, at the beginning of the 

V 

:r F | processing process of Fig. 2, i.e. before the image 

20 sharpening. 

During the taking of a digital image it can occur that 
(because of technical limitations of the camera, incorrect 
adjustments by the photographer, and so on) a generally 
underexposed image is created. In that case, the available 

25 level control (for example 0 to 255) is only insufficiently 
used for all three colours. The image thereby appears too dark 
and too flat. 

Amateur photographs are often also created under 
unfavourable conditions (incandescent light, neon light, and 
30 so on) . In such cases, the available level control for at 
least one of the three colours is insufficiently used. Such 
images have an annoying colour tinge. 

Such exposure errors and illumination errors are 
preferably corrected already at the beginning of the 
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processing chain (which means before the application of the 
correction masks, such as the above described additive and 
mutliplicative masks) . 
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